Skip to content

fix(codecs): support Avro Date and TimeMillis logical types - #26112

Open
omwbennett wants to merge 5 commits into
vectordotdev:masterfrom
omwbennett:support-avro-date-timemillis
Open

fix(codecs): support Avro Date and TimeMillis logical types#26112
omwbennett wants to merge 5 commits into
vectordotdev:masterfrom
omwbennett:support-avro-date-timemillis

Conversation

@omwbennett

Copy link
Copy Markdown
Contributor

Summary

Support encoding and decoding Avro Date and TimeMillis values.
Due to the way VRL stores integers, an extra step was added to the avro encoder path to coerce Avro types based on the schema field's logical type.

References

Closes #24773

Vector configuration

sources:
  kafka_avro:
    type: kafka
    bootstrap_servers: "kafka:9092"
    group_id: "vector"
    topics:
      - test_topic
    decoding:
      codec: avro
      avro:
        schema: |
          {
            "type": "record",
            "name": "test",
            "fields": [
              {
                "name": "date_field",
                "type": { "type": "int", "logicalType": "date" }
              }
            ]
          }
        strip_schema_id_prefix: false

sinks:
  console_out:
    type: console
    inputs:
      - kafka_avro
    encoding:
      codec: json

And similar for TimeMillis.

How did you test this PR?

Manual testing with the above configs, added unit tests for coerce_logical_types and enabled the existing date/timemillis avro round trip tests. All tests passing.

Is this a breaking change?

  • Yes
  • No

Does this PR include user facing changes?

  • Yes. Please add a changelog fragment based on our guidelines.
  • No. A maintainer will apply the no-changelog label to this PR.

Contributor Guidelines

  • Please read our Vector contributor resources.
  • Do not hesitate to use @vectordotdev/vector to reach out to us regarding this PR.
  • Some CI checks run only after we manually approve them. To catch issues early, add a pre-push hook (template) or run the following locally before pushing:
    • make fmt
    • make check-clippy (auto-fix with make clippy-fix)
    • make test
  • After a review is requested, please avoid force pushes to help us review incrementally.
    • Feel free to push as many commits as you want. They will be squashed into one before merging.
    • For example, you can run git merge origin master and git push.
  • If this PR introduces changes Vector dependencies (modifies Cargo.lock), please
    run make build-licenses to regenerate the license inventory and commit the changes (if any). More details on the dd-rust-license-tool.

@omwbennett
omwbennett requested review from a team as code owners August 14, 2026 13:12
@github-actions github-actions Bot added docs review on hold The documentation team reviews PRs only after a PR is approved by the COSE team. domain: external docs Anything related to Vector's external, public documentation labels Aug 14, 2026

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 0ca77c6bfb

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

Comment thread lib/codecs/src/encoding/format/avro.rs Outdated
Comment thread lib/codecs/src/encoding/format/avro.rs
@pront
pront enabled auto-merge August 17, 2026 16:58
@pront
pront added this pull request to the merge queue Aug 18, 2026
@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Aug 18, 2026
@pront
pront enabled auto-merge August 18, 2026 20:14

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 4808acf3b0

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread lib/codecs/src/encoding/format/avro.rs Outdated
@pront pront added the meta: awaiting author Pull requests that are awaiting their author. label Aug 18, 2026
auto-merge was automatically disabled August 19, 2026 06:51

Head branch was pushed to by a user without write access

@github-actions github-actions Bot removed the meta: awaiting author Pull requests that are awaiting their author. label Aug 19, 2026

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 9b6ef3c523

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

other => other,
};
match coerce_logical_types(value.clone(), variant, names) {
Ok(coerced) if coerced.clone().resolve(resolved_variant).is_ok() => {

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Resolve union branches with the complete name table

When a union branch is a named record that itself references another named record defined elsewhere, and that nested record contains a date or time-millis field, coercion follows the name table successfully but this guard validates the result against only the dereferenced outer record. That isolated schema cannot resolve the sibling named type, so the coerced branch is discarded; the original Long survives until full-schema resolution and encoding fails. Fresh evidence beyond the earlier fixed nullable-record case is that resolved_variant dereferences only one level, while the new regression test has no nested named reference.

Useful? React with 👍 / 👎.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

To fix I think it would be best to use avro-rs 0.22.0's resolve_with_names(). I'll open a separate PR to bump to 0.22.0 then rebase this PR and update to use the new method.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

docs review on hold The documentation team reviews PRs only after a PR is approved by the COSE team. domain: external docs Anything related to Vector's external, public documentation

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Avro decoder doesn't support Date values

2 participants